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ABSTRACT 



An experiment was conducted to determine if the use 
of rules to guide revisions has generalizable effects across programs 
and programers. The hypothesis was that pupil achievement would be 
greater for instructional programs revised according to specified 
rules than for unrevised programs. Each of 10 programers produced a 
self-instructional sequence for fifth graders. The 50-frame programs 
including pretest and 10-item posttests were to take approximately 
one hour to complete. After each was administered to two fifth 
graders the first draft programs with technical performance data were 
randomly distributed to the programers with a set of rules for use in 
revision. Then 64 fifth graders were randomly assigned to receive 
either a first draft or revised program. Scores were averaged for 
each condition, first draft or revision, for each program and the 
mean for each program per condition provided the unit of analysis. A 
t test based on the paired observation model was employed using the 
S*s mean performance per condition, matched by program, as the entry. 
An omega square value to test the strength of association was 
computed. It was inferred from the results that empirical revision 
based on rules pays off modestly, i.e., instructions can be provided 
so that the efforts expanded in revision will have some effect on the 
obtained levels of pupil performance. Confidence limits for the 
difference were calculated so that cost of revision could be 
assessed. [Not available in hardcopy due to marginal legibility of 
original document. ] (JS) 
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EVA h. BAKER 

Grasuate School of Ediica£i<,'n 
University of Cailfomlaj Los i ■>geles 



Educational practz.ee is increasing its dependence upon replicable 
instructional products for use in the schools. Without \ 7 aitlng for re- 
search findings to filter haphazardly into teaching routines, products 
can substantially alter instruction by translating research into icaiied- 
raus appiicatz-on and solution to a variety of learning problems* Repli- 
cable products derive their impact in that they are based upon learning 
principles analyzed in laboratory settings* For example- the use of con- 
iXTisation, practice and prompting is frequently observed- and each of these 
principles has a groviing literature of support. 

A primary tenet of product development (llarkle, 1967) is the employ- 
ment of empirical revision for program improvemento The use of revision 
assumes that effects of the usual first draf t effort of a given program will 
not be sufficient to allow the student masteryc validation of the character- 
istic iterative of revising practice has not been adequately studied* Silber- 
man, et al. , {196^} investigated factors associated T-rith program revision pro- 
cesses and identified particular attributes of program revisions related to 



o 



EMC 



pupil learning- Sobeek (1965) found that developmental testing improved 



the effects of a subsetjuent draft of a program- Cropper and Lisasdaine 
(1961) conducted an early study in v^ich changes based on data from a 
first trial resulted in improved performance on a revision. A diffle 
cuity ^?lth the investigations is that they lack generaiizacillty of ef- 
fect- In most cases, the investigators themselves made the revisions 
on relatively Intuitive grounds- Effects may be attributed to artistry 
rather than the po-fcr of a variable- Even ^vhere the revision rules were 
eventually articulated, as In Silberman*s study, there was no test of the 
applicability of the rules to other situations and programs - 

in order to determine if the use of rules to guide revisions has 
generalizable effects across programs and programmers, an eiq^eriment 
was conducted- The hypothesis was that pupil achievement would be greater 
for instructional programs revised according to specified rules than for 
unrevlsed programs- 

EB0CE9URE 

Developmental Ehase 

Ten programmers involved In a graduate course in programmed instruc- 
tion formed the sample- Each progrannner produced a self-instructional 
sequence designed to be administered to a child at the fifth grade levei- 
Progranaaers were told to develop their programs in accordance with instruc- 
tion (Markle, 1964) received in class, providing practice, confirming, re- 
sponse and so on- Each program was to include at least 50 frames and take 
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approximately an hour to coapieteo Any topic might be chosen by the pro- 
grsBBLBTSy altiiough directors were given to select objectives which required 
application of principles rather than simple recall. Each programmer was 
also required to prepare a pretest measuring stated entry and enroute be- 
haviors, and a posttest consisting of at least ten items o Ten first draft 
2 

programs were produced and each was individually administered to two dif^ 
ferenfc fifth grade students c Data from this developmental testing were 
sumoarized, and the report submitted by each programmer included entry 
shills, error rates, and criterion performance of the two learners o 

Revision Treafcnent 

First draft programs with technical performance data were then ran- 
do3dy distributed to the programmers; no progpcamraer received his own program 
to reviseo A set of rules was provided for the programmers to use in attempt- 
ing a revisicno Samples or such rules are listed belowi 

Ao fe'HEH EBHOR RATES ARE LOW ARS CRT.TERIOH FERFOBMAKCS POOR: 

X« Chech to see that the desired behaviors are practiced 
in the program. 

ACTION; ADD PRACTICE 

2o If the behaviors are practiced, check to see that the 
frames are not overprompted and that prompts are faded, 
so that practice in the criterion behavior is providedc 
ACTION: FADE PROMPTS 



^Topics of the programs were: Se nsory Imagery , Roles and Role Conflict , 
good Teeth Care, Reflexive PronouTrSv . H ebrew , Mge^stion, Advertisiiuz 
Technioues, Navigation , Simile and Metaphor a nd Rules for Tutors. 
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3* If the behaviors are practiced and proi^ts faded, check 
to see that difficulty of content appears to be equiva- 
lent in both program practice and criterion test* 

ACTION: mOVim PRACTICE lilTH EQUIVALENT DIFFICULTY 

B. WHEN ERSCR RATE IS HTGK AND CRITERIOil FE?xFOIU£^CE POOR: 

lo Re-analyze the task into component skills to make sure 
that en route and entry behaviors are not omitted, 

ACTION: ADD FEAI4ES TO PROMOTE MASTERY OF RELEVANT 

SUBTASKS, 

2, Hake sure student has adequate directions for the tasks 

expected. Add instructional cues, or statements of the 
rule guiding the kind of response the student is to make, 
AC-nOH: ADD INSTRUCTIONAL CUE 

ADD STATEliEKT OF INSTRUCTIONAL OBJECTIVES AT 
OUTSET OP PROGRAM 

3, Review program to determine if the programmer has at- 
tempted to ’*mo£ivate** the learner, 

ACTION: ADD ”P£INFORCSPvS” DIRECTLY TO THE PROGRAM OR 

AS A CONTINGENCY 

Programmers were told that revisions would subsequently be tested. 
Experimental Phase 

An experiment was conducted to assess the differential learner 
performance of fifth grade students subjected to first draft or re- 
vised programs. 

Sixty-four students in fifth grade classes at two schools 
were the subjects of the experiment. Subjects were randomly assigned to 
receive either a first draft or revised program. At least three children 
were treated by either version of each program. 

Procedure : Students were assembled in a large classroom and pro- 

grams were distributed at random. Following a forty minute instructional 
period, the children were posttested on the behaviors taught in their 
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prograioso Children exposed to programs on the same topic, e.g«, sen- 
sory imagery- received the same posttest o 

Criteri<m ±es t ; Each progrannaer had been directed to supply at 
least ten criterion items to serve as a posttest„ When the test sup- 
plied exceeded ten items, superfluous items i?ere randomly discarded 

from the test, using a table of random numbers, A single criterion test 
■5BS used for each pair of programs, 

^alysis: The scores mere averaged for each condition, first draft 

or revised, for each program. The man for each program per condition 
provided the unit of analysis. Programs which reached a 90% criterion 
level in the first draft form were dropped from the analysis, since there 
would be no reason to revise them. Three programs, mth mean perfo^nances 
of 97%, 100%, and 90% were thus deleted. A t test based on the paired ob- 
servation model was employed using the S»s mean performance per condition, 
matched by program, as the entry, Bata are reported in Table I below: 

Insert Table X about here 

As recommended by Hays (1963) an omega square value to test the strength of 
association was computed 

Insert Table II about here 

The relatively large omega square value might be expected given the 
sample size so small. From these results one could infer that empirical 
revision based on rules pays off modestly, that is, instructions can be pro- 




vided so diat che efforts expended in revision vTiil have some effect on 
the obtained levels of pupil perfomanceo Since the size of the dif- 
ference attributable to revision is critical to assess the cost of 
revision, the confidence limits for the difference were calculated. 

Insert Table III about here 

Sven though the absolute size of the difference is itself small, 
the difficulty of attaching a value to any increment of pupil learning 
must be faced. By converting the difference to standard deviation units, 
one can see that the mean increment obtained is equal to approximately 
-68 standard deviation units on the first draft distribution c If such 
a stable improvement could be predicted across subject matter fields, 
based on the application of these or a similar set of rules, the use of 
revision cycles could be evaluated in terms of their cost- 
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2 Table XI 

for Strength of Agsoelatlon 

»1 • 6 

» 6 

^ « 4,464 
^ « .5928 
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Table III 

95 % Confidence Intgrval for the Dlfferetice 




1.63 



% t 

(95/256) 

.365 2,447 
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